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Abstract. The effective design and use of Automated Writing Evaluation (AWE) 
tools in developing English as a Foreign Language (EFL) learners’ writing skill and 
learner autonomy have remained great challenges for system designers, developers, 
and EFL instructors compared with that of the pencil-paper writing in the context 
of regular teacher-fronted classroom in Chinese higher educational institutions. The 
function of Corrective Feedback (CF) provided by the Pigai system, i.e. a web- 
based AWE tool in China (http://www.pigai.org/), can be reflected on the students’ 
writing scripts. To measure if AWE-based CF is beneficial to EFL learners in 
improving their writing skill as well as learner autonomy, a study was carried out 
based on the Pigai system in a web-based college English course which focuses 
on developing learners’ output abilities. The correlated data and feedback showed 
that the application of AWE-based CF, i.e. 10-minute online writing performance, 
did have a marked impact on EFL learners’ writing skill and learner autonomy. In 
this paper, the effects of using the AWE system and its CF on students’ efficacy in 
English writing as well as its pedagogical effectiveness will be discussed. Learners’ 
perceptions toward AWE-based CF and the effective use of AWE tools in EFL 
contexts will also be covered. 
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1. Introduction 

CF, as defined in Wikipedia (n.d.), is frequently practiced in the field of education 
as well as learning; “[i]t typically involves a student receiving either formal or 
informal feedback on his or her performance on various tasks by a teacher or 
peer(s)” (para. 1). Lightbown and Spada (1999) stated that CF is any indication 
to the learners that their use of the target language is incorrect. Though the 
CF in pencil-paper writing class used to have significant effects on improving 
learners’ writing skills, there still exists potential problems that should not be 
ignored. 

Nystrom (1983) discussed teachers’ role when giving CF in pencil-paper style 
and he concluded that “teachers typically are unable to sort through the feedback 
options available to them and arrive at the most appropriate response” (p. 170). 
Van Lier (1988) considered that teachers had too much control of students’ 
self-initiated repairs which in turn hampered learners’ self-correction ability. 
Chaudron (1988) and his colleagues found that in traditional classes CFs were, 
to some extent, biased. 

With the rapid development in the fields of information technologies and natural 
language processing in last few decades, more powerful AWE tools have been 
developed, making it possible to release instructors’ working load while inspiring 
learners’ sense of writing and promoting their awareness of self-correction in 
writing. This study aims to investigate the effects of the CF function of the Pigai 
system on students’ writing scripts, a 10-minute online writing as an embedded 
task in an integrated college English course and its impact on the learners’ writing 
skill and learner autonomy. 

2. Method 

2.1. Research questions 

The research is driven by the following questions: 

• Is AWE-based CF helpful to improve EFL learners’ writing skills? 

• Is there a significant difference between male and female learners in 
developing their writing skills by using the AWE tool? 

• How do EFL learners perceive AWE-based CF in the process of writing? 
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2.2. Research design 

This study was carried out at one of the first author’s college English classes, in 
which there were 25 non-English majors in a digital lab from September, 2014 to 
January, 2015. All the students had passed CET-4 4 before taking the course, and 
23 students had accomplished all of the ten writing tasks throughout the semester. 

Each time in class, students were assigned to a task of four-round productive 
activities on a controlled topic: 

• Round one: 10-minute 3 -person group discussion. 

• Round two: 5 -minute pair- work. 

• Round three: 10-minute online writing. 

• Round four: 1 -minute recorded personal statement. 

In round three, students were required to write a passage about 200 words to the 
same topic on the Pigai system within ten minutes. After submission, students 
could receive their scores immediately with detailed real-time CF and comments. 
Therefore, students could revise their texts as many times as they wanted within a 
time period (set by the teacher) and their performance of each time revision could 
be tracked (see Figure 1 and Figure 2). 

Figure 1 . A screenshot of the online writing assignments on the Pigai system 



4. CET is the abbreviated form of “College English Test”. The national College English Test Band Four (CET-4) in 
China aims to evaluate non-English majors’ comprehensive language proficiency. Apart from CET-4, there is also 
CET-6, which is widely used to evaluate above-average college students’ English language proficiency. 
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Figure 2. A screenshot of the tracking record of a student’s writing 



8 2014 - 12-10 12 : 05:09 74.5 

9 2014 - 12-10 12 : 05:38 76 

10 2014 - 12-10 12 : 06:00 76 


My favorite festival is the Middle Autum Autumn Festival full of happyness.It's happiness. 
It's a traditional Chinese holiday that individuals are getting toghter to observe the festival 
and it's a fascinating thing to appreciate the moomlight as well as enjoy the 


2.3. Instruments 

One follow-up questionnaire and a scale of students’ perception toward AWE CF 
(at 5 -point Likert) were designed and delivered to the class at the end of the term. 
A complete set of 10-minute writing tasks including ten writing tasks were required 
to be completed on the Pigai system in class. 

2.4. Data collection 

All the students’ writing scripts were automatically scored and collected through 
the Pigai system. 

2.5. Data analysis 

Data was processed by using SPSS 22.0. 

• Paired sample t- tests were used to find out if there were significant differences 
between the students’ improvement in writing and their revision times. 

• Paired sample /-tests were used to find out if there were significant differences 
between male and female students in the improvement of their writing and 
their revision times. 
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Descriptive statistics (mean, standard deviation, percentage) were employed 
to analyze the students’ perception of AWE-based CF with respect to the 
10-minute online writing task. 
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3. Discussion 

3.1. Analysis of scores 

As shown in Table 1, there were significant differences in students’ writing scores 
between their first and last writing tasks, and also between their first and final 
revisions of the two writing tasks. 

Table 1. Paired sample *-test for the improvement of scores (201409-201501) 


Whole class 

Mean 

SD 

Std. Error Mean 

t 

df 

Sig. (2-tailed) 

First revision 

-8.087 

8.176 

1.704 

-4.743 

22 

.000 

Final revison 

-14.152 

9.139 

1.905 

-7.426 

22 

.000 

Revision times 

-8.087 

9.361 

1.951 

-4.143 

22 

.000 

Improvement 

-6.065 

9.132 

1.904 

-3.185 

22 

.004 


(N=23) 


3.2. Analysis of different genders 

The result of a paired sample *-test of scores for male and female students showed 
that there were significant differences across genders in the ten writing tasks. 
Besides, the revision times were increased significantly throughout the whole 
term. 

The difference was that female students, with a level of significant value of 0.046 
(*=2.252, df=ll, p<0.05), improved more significantly than male students at level 
of 0.052 (*=-2.202, p>0.05). 

3.3. Analysis of students’ perceptions 

With respect to how students perceive AWE-based CF in developing their writing 
skills in a 10-minute online writing task, nine highly correlated items from the 
questionnaire (#01, #02, #04, #11, #23— #27) were selected and the mean scores of 
the items ranged from 4.05 to 4.57, which showed that most students held positive 
attitude toward AWE-based CF and felt it helpful to improve their writing skills as 
well as learner autonomy. 
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4. Conclusions 

This study leads to the following conclusions: 

• AWE-based CF has positive effects on improving students’ writing skills, 
which can be observed from the following aspects: the scores for the first and 
final submissions increased; the improvement in the students’ writing skills 
highly correlated with their revision times; the improvement in students’ 
writing after self-revision has been made greatly. 

• Both male and female students improved their writing skill significantly in 
their first and last writing tasks, however, female students’ writing skill has 
improved more significantly than that of male students. 

• AWE-based CF, to some extent, may help EFL learners to improve their 
writing skills as well as learner autonomy. 
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